Robustness Testing of A Distributed Simulation Backplane Masters Thesis

Creating robust software requires quantitative measurement in addition to careful specification and implementation. The Ballista software robustness testing service provides exception handling measurements for a variety of application domains. This thesis describes Ballista testing of the High Level Architecture Run-Time Infrastructure (HLA RTI), a general-purpose distributed simulation backplane developed by the Defense Modeling and Simulation Office that has been specifically designed for robust exception handling. While more robust than off-the-shelf POSIX operating systems, the RTI had normalized robustness failure rates as high as 10.2%. Non-robust testing responses included exception handling errors, hardware segmentation violations, “unknown” exceptions, and task hangs. Additionally, testing repeatedly crashed one version of the RTI client through an RTI service function call. Results obtained from testing the same version of the RTI on two different Unix operating system platforms demonstrate some difficulties in providing comparable exception handling coverage across platforms, suggesting that the underlying OS can have a significant effect on the way robustness failures manifest. Results obtained from testing the same RTI interface specification produced by two different development teams illustrate common robustness failures that programmers can overlook. Testing the RTI led to scalable extensions of the Ballista architecture for handling exception-based error reporting models, testing object-oriented software structures (including callbacks, pass by reference, and constructors), and operating in a state-rich, distributed system environment. Robustness testing has been demonstrated to be a useful adjunct to highquality software development processes, and is being considered for adoption by the HLA RTI verification facility. The results of this testing yield insights into the types of robustness problems that can occur with an application specifically designed to be highly robust.